After completing this lesson, you’ll be able to:
FME Flow has a model-driven architecture because its processes are expressed as models. In FME, these models are better known as workspaces.
Workspaces are created – we call it “authored” – using FME Form. In particular, the FME Workbench application is used. FME Workbench is a client of FME Flow, and so they form a client-server pair. However, both share the same core engine and process data in the same way.
Servers are applications that provide a resource or service.
Clients make requests of servers to access a resource or service.
Together, a client-server pair form a distributed application structure. Usually, the client and server exist on separate hardware and communicate over computer networks, but sometimes both the server and client exist on the same system.
Let's make sure you get the terminology right. FME Form is the desktop program that includes the FME Workbench, FME Data Inspector, and FME Quick Translator applications. Workspace authoring occurs in the application called FME “Workbench,” but the process defined in the canvas window of FME Workbench is called a “workspace.”
Because FME Workbench is a client of FME Flow, it may be used to transfer authored workspaces to and from FME Flow. We call this transfer publishing.
FME Workbench has the ability to:
Workspaces are stored on FME Flow in devices called repositories. Each FME Flow may have multiple repositories, but any workspace can only belong to one of them, without creating a duplicated workspace.
A repository consists of two parts:
1) The .fmw files from any workspaces that have been published to FME Flow are stored in the file-based part of the repository.
2) Metadata related to the workspace is held separately in the FME Flow Database. This metadata includes information about the contents of the workspace; for example, source and destination datasets, workspace feature types, and published parameters.
Repositories are managed by the FME Flow Core. They can be accessed through the FME Flow Web Interface in the Workspaces tab.
Security in FME Flow is very important, especially controlling access to workspaces and repositories. You can think of each repository as being like a folder on a file system, with the same ability to grant access rights to individuals and groups. So, for each repository you create, be sure to check the security permissions. If you don’t, then end-users may not get access to the repository and workspaces in it!
The functionality for publishing or downloading workspaces is accessed in FME Workbench either through the menubar:
... the toolbar:
... or the Start Tab:
Workspaces can also be uploaded and downloaded directly through FME Flow on the Workspaces page:
The Publish to FME Flow tool in FME Workbench opens the simple wizard interface. The first section of which defines the FME Flow connection to use.
Selecting Connect to FME Flow opens a small window with parameters to define connection credentials. These connection details are saved so that they can be reused in the future simply by picking from the drop-down list:
The next section defines the repository in which to store the workspace:
Either an existing repository can be used, or a new one can be created. The workspace name can also be edited, even making it different from what it is saved as locally.
This publishing step only appears when there are databases and/or web connections used in the workspace that can be uploaded to FME Flow with the workspace.
This workspace contains both a database connection and a web connection that need to be uploaded to function on FME Flow.
Web connections require authentication in FME Flow in addition to adding the web service to FME Flow. Database connections require no further authentication, but care must be taken not to accidentally overwrite an existing database connection with the same name that might already be defined on FME Flow.
The final section of the publishing wizard defines which service(s) the workspace is to be registered with. A workspace may be registered for use with any number of these services.
The Job Submitter service allows FME Flow to run a workspace as-is. This is the closest to running a workspace in FME Workbench. All inputs and outputs are defined in the workspace, so data is simply written out as specified in the writers and not streamed or delivered in any other manner.
Job submission is ideal for testing workspaces, writing to databases, and running large-scale and batch translations that make use of the server process queue.
You can learn about the other services in the FME Flow Services documentation or the Build Basic Self-Serve Workflows course.
Once a workspace has been published, the republish tool becomes active. Further updates to the workspace (within the same FME Workbench session) can then be uploaded with a single click.
The same parameters are used as before. If changes need to be made to these parameters, then the Publish option going through the entire publishing wizard should be used.
Workbench can also download a workspace held in an FME Flow repository. This is usually done to make edits to a workspace already published to FME Flow. Downloaded workspaces are copies of the original, which remains in the FME Flow repository, so the workspace should be published back to FME Flow once the edits are made.
The two parts of the downloading wizard are the FME Flow connection, which is the same as the publishing wizard, and the repository and workspace selection tool:
The user is then prompted for a location to save the workspace. The default (on Windows) is <User>\Documents\FME\Workspaces. The workspace and any resources are then downloaded and saved to that location.
Once downloaded, the workspace is automatically opened within FME Workbench for editing.
Besides workspaces, it’s also possible to publish and download FME custom transformers and custom formats to and from an FME Flow repository.
When the source data for a translation is stored as files (rather than a live feed or database), it is possible to publish data to an FME Flow repository along with the workspace.
This data upload method is fast and simple but limits future access to the data. A better practice is to publish your data to the Resources folder separate from the workspace. Learn how to use the Resources folder in the Connect Data to FME Flow course.
In this workspace the source dataset is MapInfo TAB:
A MapInfo TAB dataset is made up of a series of files (.tab, .dat, .id, .map). When this workspace is published the wizard allows us to publish the data files alongside it by simply checking the box labeled Upload data files.
FME automatically selects the files to upload based on what it thinks is necessary to run the translation. If there are other files you wish to upload, or files FME selected that you don't wish to upload, the Select Files button allows you to make changes:
This dialogue also allows you to change where the files are published using Select Location, but for now, we'll ignore that setting and go with the default of publishing to the repository.
Once the publishing wizard is complete, those files are uploaded to FME Flow in the same repository as the workspace for use when running this workspace.
